feat(llm-obs): add dataset-records reads + experiment eval-metric submit - #674
Merged
platinummonkey merged 2 commits intoJul 24, 2026
Conversation
Add three llm-obs subcommands backed by the unstable llm-obs-mcp v1
endpoints, closing the gap vs the LLM Obs MCP toolset:
- experiments events submit -> POST /experiment/ingest-events
(submit_llmobs_experiment_events; metrics/tags via --file JSON,
experiment_id from the positional arg)
- datasets records -> POST /dataset/records
(get_llmobs_dataset_records; record-ids/tags/canonical-id/
dataset-version/limit/cursor/compute-schema flags)
- datasets records-full -> POST /dataset/records-full
(get_llmobs_full_dataset_records; 1-3 record ids)
These are the tools the agent-observability auto-experiment skill needs
that pup previously lacked. Docs (README coverage table, COMMANDS.md)
updated; unit tests added for success + error paths.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the `--file` body with inline `--metrics` (JSON array) and `--tags` flags so the command mirrors the submit_llmobs_experiment_events MCP tool's argument shape, letting skills map between pup and the MCP toolset. Remains a raw pass-through: metrics is parsed only to embed it as JSON and forwarded to the same ingest-events endpoint; the server validates the payload. - experiment_id stays positional - --metrics required, --tags optional (comma-separated) - update tests + add invalid-JSON negative test Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
tillwf
approved these changes
Jul 24, 2026
platinummonkey
approved these changes
Jul 24, 2026
gh-worker-dd-devflow-36fce6
Bot
added
mergequeue-status: queued
mergequeue-status: in_progress
and removed
mergequeue-status: queued
labels
gh-worker-dd-devflow-36fce6
Bot
added
mergequeue-status: done
and removed
mergequeue-status: in_progress
labels
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds three
llm-obssubcommands backed by the unstablellm-obs-mcpv1 endpoints, closing the gap between pup and the LLM Obs MCP toolset:llm-obs experiments events submit <exp-id> --filesubmit_llmobs_experiment_eventsPOST /api/unstable/llm-obs-mcp/v1/experiment/ingest-eventsllm-obs datasets recordsget_llmobs_dataset_recordsPOST /api/unstable/llm-obs-mcp/v1/dataset/recordsllm-obs datasets records-fullget_llmobs_full_dataset_recordsPOST /api/unstable/llm-obs-mcp/v1/dataset/records-fullWhy
These are the three LLM Obs MCP tools the
agent-observability-auto-experimentskill relies on that pup previously lacked (experiment eval-metric submission + both dataset-record reads). Adding them lets that workflow run through pup.Design
events submittakes the events body (metrics, optionaltags) via--fileJSON — matching the existing pup convention for writes (create/update/batch-update).experiment_idis the positional arg and overrides any value in the file.datasets recordsexposes the read filters as flags:--record-ids,--tags,--canonical-id,--dataset-version,--limit,--cursor,--compute-schema.datasets records-fulltakes--project-id,--dataset-id,--record-ids(1–3).experiments events/datasetsgroups.Tests / checks
cargo build✅cargo test llm_obs✅ — 6 new tests (success + error paths for each command)cargo clippy✅ no warningscargo fmtapplieddocs/COMMANDS.md🤖 Generated with Claude Code